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1. Introduction and Main Result 

Let [p] = {1, . . .,p} denote a totally ordered alphabet on p letters, and let a = 
) G [pi] m , P = (Pi, . . . , P m ) G [p2] ni - We say that a is order-is omorphic 
to P if for all 1 ^ i < j ^ m one has oti < ay if and only if Pi < Pj. For two 
permutations it G S n and r G Sk, an occurrence of r in tv is a subsequence 1 ^ i\ < 
%2 < ■ ■ ■ < ik n such that (tt^, . . . , ir ik ) is order-isomorphic to r; in such a context 
r is usually called the pattern. We say that % avoids r, or is r- avoiding, if there is no 
occurrence of r in it. Pattern avoidance proved to be a useful language in a variety 
of seemingly unrelated problems, from stack sorting [Kn, Ch. 2.2.1] to singularities 
of Schubert varieties [LS]. A natural generalization of single pattern avoidance is 
subset avoidance; that is, we say that it G S n avoids a subset T C Sk if tt avoids any 
r G T. A complete study of subset avoidance for the case k = 3 is carried out in 
[SS]. For k > 3 situation becomes more complicated, as the number of possible cases 
grows rapidly. Recently, several authors have considered the case of general k when 
T has some nice algebraic properties. Paper [BDPP] treats the case when T is the 
centralizer of k — 1 and k under the natural action of Sk on [k] (see also Sec. 3 for 
more detail). In [AR], T is a Kazhdan-Lusztig cell of Sk, or, equivalently, the Knuth 
equivalence class (see [St, vol. 2, Ch. Al]). In this paper we consider the case when 
T is a maximal parabolic subgroup of Sk- 
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Let Si denote the simple transposition interchanging i and i + Recall that a sub- 
group of Sk is called parabolic if it is generated by s ^ , . . . , s i r . A parabolic subgroup 
of Sk is called maximal if the number of its generators equals k — 2. We denote by -P^ m 
the (maximal) parabolic subgroup of Si+ m generated by si, . . . , s/_i, s/+i, . . . , s/+ m _i, 
and by fi jm (n) the number of permutations in avoiding all the patterns in Pi jm . 
In this note we find an explicit expression for the generating function of the sequence 
{fl,m{n)}. 

To be more precise, we prove the following more general result. Let us denote a = 
si$2 ■ ■ ■ Sk-i, that is, a = (2, 3, . . . , k, 1), and let a be an integer, ^ a ^ k — 1 (here 
and in what follows k = l + m). We denote by fi' m (n) the number of permutations in 
S n avoiding the left coset a a Pi :m ; in particular, fi m (n) coincides with fi jm (n). Let 
Fi m {x) denote the generating function of {/ ; a m (n)}, 

Recall that the Laguerre polynomial L%(x) is given by 

1 cl n 
L«(x) = -e*x-«—(e-*x^), 

and the rook polynomial of the rectangular s x t board is given by 

R s , t (x) = s\x s L t - s (-x- 1 ) 

for s < t and by R Sjt (x) = R tyS {x) otherwise (see [Ri, Ch. 7.4]). 
Main Theorem. Let A = min{/,m} ; n = max{/, m}, then 

A-1 r ( l )( m ) fi-X-1 / _ _1\ 

^%(^u-^) = E^ ! E(- 1 ) jA t^ + (- 1 ) Va! E I ' 

r=0 j=0 \j) r=0 ^ ' 

or, equivalently, 

k ~ x (-U x r» A_1 A ( l )( m ) 

^) = E^- -^-^ E( fe +^ E (-i)'W. 

where k = l + m = \ + fi. 

The proof of the Main Theorem is presented in the next section. 

As a corollary we immediately get the following result (see [Ma, Theorem 1]). 
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Corollary 1.1. Let ^ a ^ k — 1, then 

(k - 2)\(k - l) n+2 ~ k forn^k 
nl for n < k. 



/iViW 



Proof. Since R\^-i = 1 + (k — l)x, the Main Theorem implies 

pa (x] _ l-E*:, 3 * r (*-'--2)r! _ s^fc-jjl " 
F '*- l(a,) 1 - (* - 1)* l-^-Dx+lj 1 "' 

and the result follows. □ 

Another immediate corollary of the Main Theorem gives the asymptotics for 
fi, m ( n ) as ?i ^ oo. 

Corollary 1.2. fj* m (n) ~ c~f n , where c is a constant depending on I and m, and 7 
is the maximal root of L^~ X ; in particular, 7 ^ k — 2 + a/1 + 4(/ — l)(m — 1). 

Proof. Follows from standard results in the theory of rational generating functions 
(see e.g. [St, vol. 1, Ch. 4]) and the fact that all the roots of Laguerre polynomials 
are simple (see [Sz, Ch. 3.3]). The upper bound on 7 is obtained in [IL]. □ 

2. Proofs 

First of all, we make the following simple, though useful observation. 

Lemma 2.1. For any natural a, I, m, n such that l^a^l + m — 1 one has 
ftm(n) = Cf- a {n). 

Proof. Denote by p n and x n the involutions S n — > S n that take 12, ■ ■ ■ , ik) to 
(ifc, . . . , iii h) (reversal) to (n + 1 — ii, n + 1 — 12, ■ ■ ■ , n + 1 — i n ) (complement) , 
respectively. It is easy to see that for any Tc St, the involutions p n and x n provide 
natural bijections between the sets S n (T) and S n (pkT), and between S n (T) and 
S n (xkT), respectively. It remains to note that Pk^k^ a Pi,m = o~ l+rn ~ a Pi jm . □ 

From now on we assume that a^O, / ^l,m^l are fixed, and denote b = 
n — m + a. It follows from Lemma 2.1 that we may assume that a ^ m, and hence 
b ^ n. This means, in other words, that r G Sk belongs to a a Pi j7n if and only 
if (ri, . . . , 77) is a permutation of the numbers a + 1, . . . ,a + I. In what follows we 
usually omit the indices a, /, m whenever appropriate; for example, instead of fi m (n) 
we write just f(n). 

For any n ^ k and any d such that 1 ^ d ^ n, we denote by g n (ii, . . . , id) = 
9n i m(^u • • • > *d) the number of permutations 7r G S n (a a P^ m ) such that 7Tj = ij for 
j = 1, . . . , d. It is natural to extend g n to the case d = by setting g n (0) = f(n). 

The following properties of the numbers g n (ii, ■ ■ ■ ,id) can be deduced easily from 
the definitions. 
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Lemma 2.2. (i) Let k and 1 < % ^ n, then 



g n (...,i,...,i,...) = 0. 



(ii) Let n ^ k and a + 1 ^ ij ^ b for j = 1, . . . , I, then 



g n {h,---,ii) = 0. 



(iii) Let n ^ k, 1 ^ r ^ d ^ I, and a + 1 ^ ij ^ b for j = 1, . . . , d, j ^ r, then 



Proof. Property (i) is evident. Let us prove (ii). By (i), we may assume that the 
numbers are distinct. Take an arbitrary tt E S n such that tvj = ij for 

j = 1, . . . ,1. Evidently, for any r ^ a there exists a position j r > I such that 7r Jr = r; 
the same is true for any r ^ 6+1. Therefore, the restriction of tt to the positions 
1,2,...,/, J1J2, • • • , ja, jb+i,jb+2, ■ ■ ■ , jn (in the proper order) gives an occurrence of 
t G a a Pi :m in tt. Hence, tt S n (a a Pi >rn ), which means that g n (ii, ■ ■ • , k) = 0. 

To prove (iii), assume first that 1 ^ r ^ a. Let tt G S n and ttj = ij for j = 1, . . . , d. 
We define tt* G >S n -i by 



We claim that tt G S n (a a Pi jm ) if and only if tt* G ^^(cr"!^™). Indeed, the only 
if part is trivial, since any occurrence of r G a a Pi jm in tt* immediately gives rise to 
an occurrence of r in tt. Conversely, any occurrence of r in tt that does not include 
i r gives rise to an occurrence of r in tt*. Assume that there exists an occurrence of r 
in tt that includes i r . Since r ^ d ^ I, this occurrence of r contains a entries that are 
situated to the right of i r and are strictly less than i r . However, the whole tt contains 
only a — 1 such entries, a contradiction. It now follows from (1) that property (iii) 
holds for 1 ^ i r ^ a. The case b + 1 ^ i r ^ n is treated similarly with the help of 
transformation (tt G S n ) i— > (/T° G S n -i) given by 





for 1 ^ j ^ r — 1, 

for j ^ r and 7rj + i > z r , 

for j ^ r and < z r . 



(1) 




for 1 ^ j ^ r — 1, 

for j ^ r and > i r , 

for j ^ r and 7rj+i < z r . 



□ 
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Now we introduce the quantity that plays the crucial role in the proof of the Main 
Theorem. For n ^ k and 1 ^ d ^ I we put 

b 

A(n, d) = A a l m {n, d) = Y g n {h, . . . ,i d ). 

As before, this definition is extended to the case d = by setting 

A(n, O) = g n (0) = f(n). 
Theorem 2.3. Let n ^ k + 1 and 1 ^ d ^ I — 1, then 

A(n, d + 1) = a(n, d) - (m - d)A(n - 1, d) - cL4(n - 1, d - 1). (2) 

Proof. First of all, we introduce two auxiliary sums: 

6+1 

B(n,d) = Bf m (n,d) = Y g n (h, . . . ,i d ), 
C{n,d) = Cf m {n,d) = Y 9n(h, ■ ■ ■ 

i x ,...,i d =a 

once again, B{n, 0) = C(n, 0) = f(n). 

Let us prove three simple identities relating together the sequences {A(n,d)}, 
{B(n,d)}, {C(n,d)}. 

Lemma 2.4. Let n > k and 1 ^ d ^ Z ; then: 

A(n, d) = A(n, d-l)-(m- a)B(n - 1, d - 1) - aC(n - 1, d - 1), 
(m — a)A{n, d) = (m — a)B(n, d) — (m — a)dB(n — 1, d — 1), 
aA(n, <i) = aC(n, <i) — adC(n — 1, d — 1). 

Proof. To prove the first identity, observe that by definitions and Lemma 2.2(iii) for 
the case r = d, one has 

b n 

A(n, d - 1) - A(n, d) = Y ^ g n (ii, ■ ■ ■ , id) - A(n, d) 

ii,...,id-i=a+l id = l 
b / a n 

= Y [ Y 9n(k,---,id) + Y 9n{h,---,id) 

i 1 ,...,id-i=a+l \id = l i d =b+l 

b 

= Y (ag n - 1 (i 1 -l 1 ... 7 i d - 1 -l) + (m-a)g n - 1 (i u ...,i d - 1 )) 

ii,...,id-i=a+l 

= aC(n - 1, d - 1) + (m - a)B(n - 1, d - 1), 
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and the result follows. 

The second identity is trivial for a = m, so assume that ^ a ^ m — 1 and observe 
that by definitions and Lemma 2.2(h) and (hi), one has 

b 

B(n 7 d)= ^ 0n(*i, •••,*«*) 

ii,...,i d =a+l 

d b 

+ ^2 9n(h,---,ij-i,b+l,i j+ i...,id) 

3 = 1 ii,...,ij,...,i d =a+l 
d b 

= A(n,d) + ^2 g n -i(ii,...,ij-i,ij+i,...,id) 

3=1 ii,...,ij,...,id=a+l 

= A(n, d) + dB(n - 1, d - 1), 

and the result follows. 

Finally, the third identity is trivial for a = 0, so assume that 1 < a ^ m and 
observe that by definitions and Lemma 2.2(h) and (hi), one has 

b 

C(n,d)= g n (ii,---,id) 

ii,...,id=a+l 
d b 

+ g n (h,---,ij-i,a,i j+ i...,id) = A(n,d) 

j=l i 1 ,...,ij,...,i d =a+l 
d b 

+ ^2 ^2 9n-i{h - 1, •• -,ij-i ~ Mj+i - 1, •• -,id ~ 1) 

j = l ji,...,ij,...,i d =a+l 

= A(n,d) + dC(n-l,d-l), 

and the result follows. □ 

Now we can complete the proof of Theorem 2.3. Indeed, using twice the first 
identity of Lemma 2.4, one gets 

A(n, d + 1) = A(n, d) - (m - a)B(n, d) - aC(n -l,d-l), 
dA(n - 1, d) = dA(n - 1, d - 1) - d(m - a)B(n - 2, d - 1) - daC(n - 2, d - 1). 

Next, the other two identities of Lemma 2.4 imply 

A(n, d + 1) - dA(n - 1, d) = A(n, d) - dA(n - 1, d - 1) 
- ((m - a)B(n - 1, d) - (m - a)dS(n - 2, d - 1)) - (aC(n - 1, d) - adC(n - 2, d - 1)) 
= A(n, d) - dA(n - 1, d - 1) - (m - a) A(n - 1, d) - aA(n - 1, d), 

and the result follows. □ 

The next result relates the sequence {A(n,d)} to the sequence {/(n)}. 
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Theorem 2.5. Let n > k and 1 ^ d ^ I, then 

d 



n-j). 



Proof. Let -D(n, d) = Df m (n, d) denote the right hand side of the above identity. We 
claim that for n ^ k + 1 and 1 ^ d ^ I — 1, D(n, d) satisfies the same relation (2) as 
A(n, d) does. Indeed, 

D(n - 1, d) = (™) Q f(n - 1 - j) 

g<-*«-<-i)G->-'> 

+ ( m _ d )(_l)d+i rf! ^ / ( n _ c /_l) ) 



and 



£>(n - 1, d - 1) = (™) J /(n " 1 " j) 



and hence 

D(n, d) - (m - d)D(n -l,d)- dD(n - 1, d - 1) 

= /(„) + ( m _ d )(-l) d+1 d! Q /(n - d - 1) 

= /(n) + ]j> W (7) I ^ /(" - J) + ( rf + ( d + i) /(" - rf - !) 

= D(n,d+1). 

It follows that D(n,d) (as well as A(n, d)) are defined uniquely for n ^ k and 
1 ^ / ^ d by initial values D(k, d), -D(n, 0), and D(n, 1) d), A(n, 0), and A(n, 1), 
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respectively). It is easy to see that for n ^ k one has A(n, 0) = D(n, 0) = f(n). Next, 
the first identity of Lemma 2.4 for d = 1 gives 

A(n, 1) = A(n, 0) - (m - a)B(n - 1, 0) - aC(n -1,0) = f(n) - mf(n - 1) for n ^ k. 
On the other hand, by definition, 

D(n, 1) = f(n) — mf(n — 1) for n ^ /c, 
and hence A(n, 1) = D(n, 1). Finally, a simple combinatorial argument shows that 

A(k, d) = d\ (A; - d)! - /!m! for 1 ^ d ^ I. 

On the other hand, 

D(k, d) = XJ( 1) ',! (™) Q (* - j)! " 

since /(r) = r! for 1 ^ r < k — 1 and /(fc) = k\ — l\m\. To prove A{k, d) = D(k, d) it 
remains to check that 

which follows from Lemma 2.6 below. □ 

Finally, we are ready to prove the Main Theorem stated in Sec. 1. First of all, by 
Lemma 2.2(h), A(n,l) = for n ^ k. Hence, by Theorem 2.5, 



l 

/ in \ i i \ 



or, equivalent ly, 

I 



x 3 f(n — j)x n 3 = for n ^ fc. 

\J / V// 

As it was already mentioned, f(r) = r! for 1 ^ r ^ k — 1, therefore 

b-^ ! (7)C)* , (^-1" , ^)-°- 



(3) 
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Recall that the rook polynomial of the rectangular s x t board, s ^ t, satisfies 
relation 



x J 



(see [Ri, Ch. 7.4]). Hence, (3) is equivalent to 



k-j-i 



FZ m {x)R Xtlt {-x) = £(-W(7) ( - V E 

k — 1 r 

= E* ?>! E(- 1 



X l\ 



k ~ 1 ' - ., 0(7) 
' 



A-l 



r=0 j=0 

V7/ r=u i=0 v/V 



£ xV! £(-l)i-£^L_ + E E 

r=0 7=0 \j) r=A 7=0 



r=M j=0 



By Lemma 2.6 below, the third term of the above expression vanishes, while the 
second term is equal to 



2- 1 ' (i-r-l)!r! ' 

r=A 



and the first expression of the Main Theorem follows. The second expression is 
obtained easily from (3) and relation between rook polynomials and Laguerre poly- 
nomials given in Sec. 1. □ 

It remains to prove the following technical result, which is apparently known; 
however, we failed to find a reference to its proof, and decided to present a short 
proof inspired by the brilliant book [PWZ] . 



Lemma 2.6. Let 1 < s t and let 



m(m) = £(-i; 



J Ai 



0(D 



i=0 



Then: 



'(V) 


I 

/s+t-n-l\ 
j\s \ s ) 



M(s,t) = < 



(-1) 







(?) ' 

if n ^ s + t, 
ift^n^s + t-1, 

if s ^ n ^ t — 1. 
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Proof. Direct check reveals that M(s, t) is a hypergeometric series; to be more precise, 

M(s,t) = 2 F 1 

Since — s is a nonpositive integer, the Gauss formula applies (see [PWZ, Ch. 3.5]), 
and we get 

»,/ \ ,. T(-z + t + s)T(-z) 
M(s,t) = lim \ -|~ ■ / N ■ 

V ' z^n F(t-Z)T(S-Z) 

Recall that 

r(x)T(l-x) = —^—. (4) 
sin7rx 

If n ^ s + 1, we apply (4) for a; = — z + t + s, x = t — z, x = s — z, x = —z, and get 

= T(n-t + l)T(n-s + l) ^ smir(t-z) sinTr(s-z) = (V) 
{,) T(n-t- s + l)r(n+l) z^n smnzsmTi(t + s-z) ~~ (™) 

If £ ^ n ^ s + £ — 1, we apply (4) for x = t — z, x = s — z, x = —z, and get 

= T(n-t + l)T(n-8 + l)r(8 + t-n) ^ sinTrft-z) sin n(s - z) = Q 
F(n + 1) sin7T2 

Finally, ifs^n^t — 1, we apply (4) for x = s — z, x = —z, and get 

_r ( , + t -n)r(n-, + l) sin^) ^.C^T 1 ), 
V ' ; r(t-n)r(n+l) *-n suittz v ; (™) 



□ 



3. Concluding remarks 



Observe first, that according to the Main Theorem, F^ m {x) does not depend on 
a; in other words, \S n (Pi j7n )\ = \S n (a a Pi jTn )\ for any a. We obtained this fact as a 
consequence of lengthy computations. A natural question would be to find a bijection 
between S n (Pi jm ) and S n (a a Pi tm ) that explains this phenomenon. 

Second, it is well known that rook polynomials (or the corresponding Laguerre 
polynomials) are related to permutations with restricted positions, see [Ri, Ch.7,8]. 
Laguerre polynomials also arise in a natural way in the study of generalized derange- 
ments (see [FZ] and references therein). It is tempting to find a combinatorial relation 
between permutations with restricted positions and permutations avoiding maximal 
parabolic subgroups, which could explain the occurrence of Laguerre polynomials in 
the latter context. 

Finally, one can consider permutations avoiding nonmaximal parabolic subgroups 
of Sk. The first natural step would be to treat the case of subgroups generated by k— 3 
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simple transpositions. It is convenient to denote by Pi lt i 2 ,i 3 (with h+h + h = k) the 
subgroup of Sk generated by all the simple transpositions except for and s^+fe; 
further on, we set fw,i 2 ,i 3 {n) = \S n (Pi 1:h:h )\, and Fi uh:h (x) = T, n ^o fh,h,h (n)x n . 
It is easy to see that i 7 / 1 ,j 3) j 3 (x) = Fi 3 ,i 2 ,h{ x )i so one can assume that l\ ^ I3. This 
said, the main result of [BDPP] can be formulated as follows: let k ^ 3, then 

*i,i,fe-2(a0 = ^ x r r\+ [ - } 2 X (l - (fc - l)x - a/1 - 2(/c - l)x + (k — 3) 2 x 2 j . 

r=l 

To the best of our knowledge, this is the only known instance of Fi lt i 2i i 3 (x). It is 
worth to note that even in this, simplest, case of nonmaximal parabolic subgroup, 
the generating function is no more rational. 
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